Ensemble Learning for Hidden Markov Models

The standard method for training Hidden Markov Models optimizes a point estimate of the model parameters. This estimate, which can be viewed as the maximum of a posterior probability density over the model parameters, may be susceptible to over-tting, and contains no indication of parameter uncertainty. Also, this maximummay be unrepresentative of the posterior probability distribution. In this paper we study a method in which we optimize an ensemble which approximates the entire posterior probability distribution. The ensemble learning algorithm requires the same resources as the traditional Baum{Welch algorithm. The traditional training algorithm for hidden Markov models is an expectation{ maximization (EM) algorithm (Dempster et al. 1977) known as the Baum{Welch algorithm. It is a maximum likelihood method, or, with a simple modiication, a penalized maximum likelihood method, which can be viewed as maximizing a posterior probability density over the model parameters. Recently, Hinton and van Camp (1993) developed a technique known as ensemble learning (see also MacKay (1995) for a review). Whereas maximum a poste-riori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution over the parameters. The objective function that is optimized is a vari-ational free energy (Feynman 1972) which measures the relative entropy between the approximating ensemble and the true distribution. In this paper we derive and test an ensemble learning algorithm for hidden Markov models, building on Neal